Port conversation-aware STT recognition - #1913
Conversation
🦋 Changeset detectedLatest commit: 48e902b The changes in this PR will be included in the next version bump. This PR includes changesets to release 35 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| await this.agentSession?._keytermDetector.close(); | ||
| this._schedulingPaused = true; |
There was a problem hiding this comment.
🟡 Scheduling pause is delayed by an async operation, allowing unintended speech scheduling during drain/handoff
The scheduling-pause flag is set (this._schedulingPaused = true at agent_activity.ts:3878) only AFTER an awaited keyterm detector close (agent_activity.ts:3877), so during that async wait new speech handles can still be enqueued.
Impact: During agent handoffs or session shutdown, extra speech handles may be scheduled and processed, extending drain time unpredictably.
Pre-PR ordering change and race window
Before this PR, _pauseSchedulingTask set this._schedulingPaused = true as its very first statement (after the early-return guard). This immediately caused scheduleSpeech() (agent_activity.ts:3862) to throw SchedulingPausedError for any new callers, preventing new speech handles from entering the queue.
The new code inserts await this.agentSession?._keytermDetector.close() before the flag is set. KeytermDetector.close() (keyterm_detection.ts:192-201) awaits any in-flight detectTask, which can block for up to the detection timeout (default 10 seconds via DETECTION_TIMEOUT at keyterm_detection.ts:42). During this window:
this._schedulingPausedis stillfalsescheduleSpeech()succeeds instead of throwing- Tool responses or concurrent
generateReply()calls can enqueue new speech handles - These handles must then be drained before the pause completes
The fix is to set the flag before the async close, or move the close after the flag is set:
this._schedulingPaused = true;
await this.agentSession?._keytermDetector.close();| await this.agentSession?._keytermDetector.close(); | |
| this._schedulingPaused = true; | |
| this._schedulingPaused = true; | |
| await this.agentSession?._keytermDetector.close(); |
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
Verification
pnpm --filter @livekit/agents build:typespnpm --filter @livekit/agents lint(passes with existing warnings)pnpm --filter @livekit/agents buildNotes
pnpm --filter @livekit/agents api:checkis blocked by the existing API Extractor limitation onexport * as ___indist/index.d.ts.Ported from livekit/agents#6039
Original PR description
Makes STT recognition conversation-aware via a new
stt_context_optionsonAgentSession, grouping static keyterms with two independent, composable mechanisms for biasing recognition during a call.Overview
keyterms: user-defined terms applied wherever the STT accepts a term list; never modified by detection.keyterm_detection: an LLM-based detector (enabled,llm,turn_interval,max_keyterms,instructions) that runs a background pass per user turn over the recent transcript and maintains the keyterm set with a confirmation gate — a new term startspendingand only biases the STT once later transcript evidence confirms it;removeapplies only to spellings the user explicitly corrected.chat_context: native conversation-context carryover for STTs that consume context directly (no LLM), forwarding each conversation turn to the provider's own field (e.g. AssemblyAI u3-rt-proagent_context).Details
STT._push_conversation_item()hook andchat_contextcapability flag, alongside the existing_update_keyterms()/keytermsflag; both are forwarded by the fallback and stream adapters.keytermscapability implemented for deepgram (v1/v2), assemblyai, google, and livekit inference STT;chat_contextimplemented for assemblyai u3-rt-pro.